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The  addition  of  network  interfaces  and fusion  algorithms  to  sensor  systems  results  in  an  increase 
in  the  complexity  of  the  required  test  and  evaluation  technologies  and  methods.  Existing  testing 
capabilities  are  not  adequate  for  testing  of  tactical  networked  sensor  hardware  in  complex 
battlefield  configurations.  We  are  developing  a  system  for  testing  the  wireless  sensor  networks 
that  support  tactical  hardware  interacting  in  real  time  with  emulated  network  nodes  in  an 
augmented  reality  test  scenario.  The  test  bed  provides  a  realistic  simulated  representation  of  a 
tactical  network  that  allows  faithful  testing  of  networked  systems  focusing  on  hardware-in-the- 
loop  testing  of  sensors  and  sensor  fusion  systems.  Systems  can  be  tested  using  this  method  in  a 
controlled,  repeatable  environment  not  feasible  in  field  testing.  The  system  design  combines 
dedicated  high  performance  computing  resources  with  a  scalable,  high  fidelity  network  emulation 
and  a  computer  generated  forces  model  to  virtually  represent  the  tactical  network,  force 
movement,  interactions,  and  communication  loads  to  systems  under  test.  This  article  presents  the 
test  bed  design  framework,  preliminary  performance  results,  and  a  concept  for  determining  the 
requirement  and  performance  envelope  for  test  bed  utilization. 
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Realistic  testing  of  tactical  wireless  sensor 
networks  requires  enhanced  techno¬ 
logies  and  techniques  utilizing  real-time 
hardware-in-the-loop  (HWIL)  test 
methods.  A  mixture  of  live  and  simu¬ 
lated  network  nodes  operating  in  real  time  and 
immersed  in  an  augmented  virtual  environment  is  the 
optimal  approach  to  obtaining  highly  accurate  test  data 
with  the  abihty  to  scale  the  network  size  to  tactical  force 
levels.  Performing  a  fully  live  field  test  is  impractical 
because  of  the  size  and  variations  of  the  terrain  required 
and  the  sheer  number  of  tactical  network  nodes  needed 
to  completely  represent  the  full  force  stmcture  and 
equipment  (i.e.,  networks,  radios,  sensors,  and  weapon¬ 
ry).  The  HWIL  test  bed  approach  described  in  this 
article  combines  dedicated  high  performance  computing 
(HPC)  resources  with  a  scalable,  high  fidelity  network 
simulation  and  a  computer  generated  forces  (CGF) 
model  to  virtually  represent  the  tactical  network,  force 
movement,  interactions,  and  communication  loads  to 


systems  under  test.  The  network  emulation  and  CGF 
models  are  required  to  interoperate  and  scale  to  the  size 
of  an  Army  brigade  combat  team  that  will  have 
thousands  of  network  nodes.  The  use  of  this  test 
method  allows  testers  to  interface  a  small  number  of  real 
hardware  nodes  with  virtual  components  to  produce  an 
operationally  reaUstic  environment. 

HWIL  testing  background 

Sensor  and  missile  systems  have  historically  relied  on 
HWIL  testing  methods  to  determine  many  aspects  of 
networked  system  under  test  (NSUT)  performance. 
The  technique  of  HWIL  testing  is  based  on  using 
actual  tactical  hardware  and  software  interfaced  to  a 
suite  of  stimulus  and  measurement  systems  including 
modeling  and  simulation  tools.  Laboratory  HWIL  test 
methods  are  considered  the  highest  fidelity  alternative 
to  live  field  testing  because  of  the  inclusion  of  the 
actual  hardware  and  software  in  the  test,  as  opposed  to 
pure  simulation  or  mathematical  models  (Almendinger 
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and  LeSueur,  2007).  To  realize  the  benefit  of  HWIL, 
we  must  ensure  that  the  NSUT  interacts  with  a  virtual 
environment  that  completely  replicates  the  real  world  if 
the  NSUT  is  to  perceive  and  respond  as  it  would  in  an 
actual  fielded  situation. 

In-band  stimulation  of  sensor  and  missile  systems 
with  visible,  infrared,  acoustic,  and  seismic  sensors  has 
become  common  place  in  the  test  and  evaluation 
(T&E)  community.  Other  forms  of  stimulation  of  the 
NSUT  to  complete  the  virtual  environment  include 
physical  motion  (pitch,  yaw,  and  roll  of  the  platform), 
launcher  or  vehicle  electric  interfaces,  and  Global 
Positioning  System  (GPS)  to  name  a  few. 

An  example  of  this  type  of  HWIL  capability  is  the 
Advanced  Multispectral  Simulation  Test  Acceptance 
Resource  (AMSTAR)  (Almendinger  and  LeSueur 
2007).  The  AMSTAR  is  unique  in  the  ability  to 
perform  real-time  multispectral  scene  generation  and 
projection  into  a  common  missile  seeker  aperture 
mounted  on  a  multiaxis  flight  motion  simulator.  The 
AMSTAR  is  equipped  with  a  dynamic  infrared 
scene  projector,  millimeter  wave  projection,  and  a 
semiactive  laser  return  simulator.  The  three  project¬ 
ed  beams  are  combined  to  provide  simultaneous 
in-band  stimulation  into  a  single  sensor  aperture 
(Figure  1). 

With  the  increase  in  complexity  of  sensors  systems 
that  include  network  interfaces,  the  HWIL  test 
methods  must  be  enhanced.  The  network  addition  to 
the  test  item  is  another  aperture  whose  input  must  be 
properly  stimulated  and  output  accurately  measured. 
Often  other  networked  systems  receive  and  respond  to 
sensor  traffic  and  changing  conditions  in  the  network 
environment.  Therefore,  simply  playing  back  recorded 
network  traffic  cannot  be  used  to  comprehensively  test 
the  functionality  of  a  NSUT. 

The  real-time  sharing  of  information  from  multiple 
sensors  across  a  wireless  network  coupled  with  the  use 
of  sensor  fusion  creates  a  “system  of  systems”  where  the 
combined  performance  has  the  potential  to  be  greater 
than  the  sum  of  the  individual  system  capabilities.  The 
T8cE  of  the  system  of  systems  must  take  into  account 
the  performance  difference  when  sensors  share  infor¬ 
mation  such  as  detections,  identifications,  moving 
object  tracks,  photos,  and  live  video  across  the  sensor 
network.  The  performance  of  the  system  of  systems 
depends  greatly  on  the  performance  of  the  network 
linking  the  individual  nodes. 

The  software  emulation  approach  to  network  testing 
is  more  cost  effective,  scalable,  and  adaptable  than 
hardware  emulation  (Werner- Allen,  Swieskowski,  and 
Welsh  2009).  The  network  simulation  provides  the 
background  traffic  present  in  tactical  situations  and 
transports  sensor  data  to  consuming  applications,  with 


Figure  1.  HWIL  to  simulation  interface. 


realistic  representation  of  radio  frequency  propagation 
and  terrain  effects,  delivery  time,  packet  loss,  collisions, 
and  bandwidth  availability.  By  providing  interfaces  to 
actual  tactical  hardware,  the  wireless  sensors  can  be 
tested  in  a  reliable  and  repeatable  way  not  available 
through  typical  field  level  test  methods. 

Proposed  testing  method 

The  core  of  the  wireless  tactical  sensor  test  bed  is  the 
high  fidelity,  real-time,  network  simulation  made 
possible  by  a  parallel  computing  platform  upon  which 
the  simulation  runs.  The  system  is  implemented  using 
EXata  network  emulation  and  OneSAF  CGF  running 
on  a  parallel  Linux  blade  system.  The  test  system 
integrates  sensors  to  the  simulated  virtual  environment 
through  wireless  gateways  and  Ethernet  connections. 

EXata  is  a  wireless  network  emulator  that  connects 
to  live  networks  and  supports  real-time  operations 
(SNT  2009).  EXata  creates  a  simulated  network  that 
interfaces  with  real  networks  allowing  for  software, 
hardware,  and  human-in-the-loop  test  applications  to 
communicate  over  all  layers  of  the  network. 

Network  emulation  with  HWIL  interfaces 

The  need  for  having  real  wireless  sensor  hardware 
connected  with  the  simulated  network  drives  the 
requirement  for  operating  in  real  time.  As  the  size 
and  complexity  of  the  modeled  network  increases,  the 
test  bed  computing  resources  must  continue  to  perform 
all  required  calculations  in  real  time  to  allow  system 
testing  (Hamida,  Chelius,  and  Gorce  2008).  When 
real-time  performance  cannot  be  maintained,  the  test 
and  simulation  results  are  considered  invalid  and 
adjustments  to  the  test  bed,  such  as  the  size  of  the 
simulated  environment  or  simulation  fidelity,  must  be 
performed  until  real-time  performance  can  be  reliably 
maintained. 


334  ITEA  Journal 


Testing  Wireless  Sensor  Networks 


Each  HWIL  device  under  test  will  have  a  corre¬ 
sponding  virtual  node  in  the  network  simulation 
engine.  This  connection  approach  is  illustrated  in 
Figure  2  where  the  example  of  a  HWIL  network  node 
is  an  unmanned  aerial  system  (UAS)  supplying  video  to 
a  network  of  simulated  nodes,  and  a  second  HWIL 
command  system  is  receiving  the  UAS  video.  The 
light-colored  nodes  correspond  to  HWIL  items  in  the 
real  environment  and  provide  the  interface  to  the 
emulated  network  represented  hy  the  hlack  nodes.  The 
video  quality  and  latency  are  affected  hy  operations  of 
the  emulated  nodes  just  as  they  would  be  tactically. 


thousands  of  heterogeneous  networked  nodes  with  a 
wide  range  of  processing  power  and  network  bandwidth 
requirements.  Both  the  network  simulation  code  and 
the  CGF  model  are  required  to  scale  to  this  magnitude. 

The  system  architecture  with  the  major  interfaces 
between  the  HPC  and  the  HWIL  interfaces  is 
presented  in  Figure  3.  The  HWIL  interface  supports 
operation  of  different  wireless  network  configurations 
including  wireless  sensors,  network  missile  systems, 
vehicles,  and  UAS. 

EXata  can  operate  in  a  shared  memory  or  message 
passing  interface  parallel  processing  environment. 
There  are  manual  and  automatic  parallel  workload 
distribution  methods.  The  manual  method  allows  the 
simulation  operator  to  assign  the  workload  associated 
with  simulated  nodes  to  computational  cores  as 
desired.  The  automatic  method  assigns  network  nodes 
to  computational  cores  in  sequential  order  as  the 
emulated  nodes  are  defined  in  the  environment.  After 
all  cores  have  one  node  assigned,  the  process  repeats 
starting  from  the  first  core  until  aU  of  the  emulated 
nodes  have  been  assigned. 


Parallel  processing  of  simulation  environment 

Network  emulation  allows  interfacing  and  testing  of  a 
few  samples  of  real  hardware  with  virtual  components  to 
produce  operationally  realistic  numbers  of  network 
nodes.  The  Army  brigade  combat  team  is  the  target 
size  for  the  development  of  the  tactical  wireless  sensor 
network  test  bed.  The  brigade  combat  team  will  have 


Test  bed  performance  results 

The  test  bed  HPC  system  is  scheduled  to  integrate  at 
the  Redstone  Technical  Test  Center  in  early  fall  2009. 
To  establish  performance  expectations  of  the  completed 
test  bed,  we  executed  initial  test  cases  on  an  existing 
parallel  computing  platform  with  an  early  release  of 
EXata  2.0.  For  these  preliminary  test  cases,  the 
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Figure  3.  System  interface  architecture. 
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Figure  4.  Parallel  execution  time  for  scenario  1. 


Figure  5.  Parallel  execution  time  for  scenario  2. 


simulation  is  executed  on  a  Silicon  Graphics  Inc. 
(Freemont,  California)  Altix  XE  320  cluster  with  gigabit 
Ethernet  connections.  One  connection  is  designated  for 
administrative  function,  and  another  one  is  for  data 
transfers.  The  cluster  nodes  consisted  of  two  Intel  Xeon 
2.5  GHz  E5420  four-quad-core  processors  (total  of 
eight  cores/node)  and  8  GB  of  RAM  per  node.  The 
system  has  a  total  of  32  processors  or  128  cores. 

Experimentation  scenario  description 

Two  existing  representative  scenarios  were  chosen 
to  be  used  in  the  test  bed  evaluation  tests.  The  two 
scenarios  were  selected  based  on  differences  in  the 
simulated  node  count  and  the  complexity  and  fidelity 
of  the  environment  model.  The  first  scenario  has  a 
node  count  of  500  radios.  Each  radio  is  mobile  and 
travels  according  to  a  group  mobility  model  that 
limits  the  movement  of  nodes  to  an  area  around  an 
established  group  location.  The  propagation  path  loss 
model  used  was  the  irregular  terrain  model,  and  an 
associated  terrain  file  was  loaded  for  use  by  the 
irregular  terrain  model. 

The  second  test  scenario  has  800  radios  simulated 
and  68%  of  the  nodes  are  mobile  with  their  movement 
defined  by  an  input  mobility  file.  A  two-ray  propaga¬ 
tion  path  loss  model  is  used  in  this  scenario. 

The  two  test  scenarios  were  selected  because  of 
their  size,  variations  in  computational  load,  and 
tactical  relevance.  The  second  scenario  has  a  larger 
node  count  but  uses  a  more  simplistic  two-ray 
propagation  model.  Because  only  a  subset  of  the 
nodes  is  moving,  the  number  of  path  loss  calcula¬ 
tions  that  need  to  be  performed  during  the  test  is 
limited. 

Preiiminary  resuits 

The  first  scenario  was  executed  and  the  simulation 
run  time  was  measured  for  a  section  of  the  scenario 
(Figure  4).  The  real-time  threshold  for  this  scenario 


section  was  360  seconds.  The  simulation  run  time  is 
plotted  with  the  number  of  computer  cores  in  Figure  3. 
The  gray  line  shows  the  measured  performance  from  1 
to  32  processing  cores  when  the  interprocessor 
communications  occur  over  the  Ethernet  interconnect. 
The  black  line  shows  the  measured  performance  when 
the  computational  node  is  allowed  to  use  a  shared 
memory  interface.  Note  that  this  data  stops  at  eight 
cores,  the  number  of  cores  on  a  node.  There  is  slight 
improvement  in  performance  when  using  the  shared 
memory  communication  versus  the  Ethernet. 

The  test  results  from  the  second  scenario  are  shown 
in  Figure  5  in  a  similar  format.  In  both  scenarios,  the 
optimum  performance  occurs  when  operating  on  eight 
computer  cores  for  the  given  size,  scenario  complexity, 
and  modeled  environment  fidelity. 

The  parallel  efficiency  of  the  two  scenarios  using 
eight  processing  cores  and  communicating  across  the 
Ethernet  interconnection  is: 

313  s 

Scenario  A  JEfficiency=  ^  *  100  =  44.5% 
8Cor« 

223  S 

Scenario  A -Efficiency  =  ^  ^  *  100  =  42.9% 

8  Cores 

Concept  of  a  performance  and  requirement 
enveiope 

As  seen  in  the  preliminary  results  section,  the  real¬ 
time  performance  of  network  emulation  capabilities  is 
a  multidimensional  problem  that  includes  the  size 
(number  of  nodes)  of  the  scenario,  the  amount  of 
computing  resources  available,  and  the  fidelity  or 
complexity  of  the  simulation  environment.  There  is  a 
need  to  establish  a  wireless  sensor  network  perfor¬ 
mance  or  requirement  envelope  to  aid  in  NSUT  test 
planning  and  resource  allocation.  This  performance  or 
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Figure  6.  Test  bed  performance  or  requirement  envelope. 


requirement  envelope  can  be  established  through  the 
collection  of  a  set  of  empirical  test  data  and  the 
development  of  a  functional  model  of  the  test  bed. 

Figure  6  shows  a  three-dimensional  plot  from  a 
notional  test  bed  configuration  and  set  of  scenarios. 
The  notional  test  scenario  real-time  threshold  is 
100  seconds.  These  data  are  used  only  to  describe  the 
utility  in  establishing  the  envelope. 

For  the  notional  case,  the  fidelity  or  complexity  of 
the  scenario  is  held  constant  while  the  number  of  radio 
nodes  and  number  of  test  bed  processors  are  changed. 
The  resulting  simulation  runtime  is  ploted  on  the 
vertical  axis.  Once  the  envelope  is  developed,  test 
configurations  can  be  determined  based  on  several 
limitations  or  requirements.  For  example.  Figure  7a 
shows  a  notional  case  where  the  number  of  computa¬ 
tion  cores  is  limited  to  four  (potentiahy  a  field  test 
where  access  to  an  HPC  is  not  available).  A  plane  is 
drawn  across  the  envelope  corresponding  to  this  test 
limitation. 

Figure  7b  shows  an  associated  graph  where  the 
processor  count  is  limited  to  four  and  the  scenario  size 
is  plotted  against  the  simulation  run  time.  The  point 
where  the  runtime  crosses  the  100-second  real-time 
threshold  establishes  that  a  maximum  of  500  nodes,  at 
the  given  fidelity  or  complexity,  can  be  emulated  on  a 
four-processor  machine  while  maintaining  real-time 
performance. 

Another  use  of  the  performance  or  requirement 
envelope  is  shown  in  Figure  8a.  In  this  case  the  test 
application  demands  a  scenario  that  simulates  800 
radios.  A  plane  is  drawn  to  show  where  this 
requirement  intersects  with  the  envelope. 

Figure  8b  shows  the  companion  graph  where  the 
scenario  size  is  set  to  800  and  the  simulaiton  run  time 
is  plotted  against  the  number  of  processing  cores.  The 
point  where  the  runtime  crosses  the  real-time  thresh¬ 
old  (100  seconds)  shows  that  16  or  more  processors  are 
necessary  to  maintain  real-time  performance  for  the 
800-node  scenario. 
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Figure  1.  (a)  Performance  enveiope  with  four-processor  plane, 
(b)  Performance  plot  using  four  cores. 


A  different  three-dimensional  graph  is  necessary  for 
each  of  the  various  levels  of  simulation  fidelity  and 
scenario  complexity  desired  for  a  set  of  test  applica¬ 
tions.  The  development  of  a  test  bed  performance 
model  validated  with  samples  of  empirical  test  data  will 
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Figure  8.  (a)  Performance  envelope  with  800-node  scenario 
plane,  (b)  Performance  plot  for  800-node  scenario. 
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allow  the  complete  suite  of  graphs  to  he  generated.  The 
performance  or  requirement  envelope  provides  an 
estimation  tool  useful  for  many  functions  including 
test  planning,  experiment  design,  and  resource  alloca¬ 
tion. 

Conclusion 

Current  test  methods  are  not  adequate  for  testing 
tactical  wireless  network  hardware  in  realistic  battle¬ 
field  environments  (LeSueur  and  Jovanov  2009).  The 
use  of  an  augmented  wireless  sensor  test  bed  operating 
in  a  real-time  HWIL  configuration  is  a  viable  solution 
to  the  T6cE  challenges  associated  with  tactical 
networked  sensors  operating  in  complex  battlefield 
configurations.  It  is  established  that  real-time  perfor¬ 
mance  improvements  can  be  realized  when  operating 
on  parallel  computer  cores  for  a  given  test  scenario. 
Through  this  evaluation,  it  is  determined  that  general 
performance  thresholds  can  be  measured,  but  the 
results  are  highly  scenario  dependent.  A  performance 
or  requirement  envelope  is  needed  to  accurately  predict 
wireless  sensor  test  bed  performance  based  on  multi¬ 
dimensional  setup  parameters. 

Follow-on  research  and  testing  are  needed  in  several 
areas  to  fully  realize  the  benefits  of  the  test  bed. 

•  Future  performance  enhancements  will  be  real¬ 
ized  when  the  simulation  engine  is  transferred  to 
the  new  HPC  cluster  with  Infmiband  intercon¬ 
nections.  The  new  system  will  have  increased 
computing  performance,  and  the  interconnect 
architecture  will  have  lower  latency. 

•  Interfaces  to  the  CGF  model  must  be  completed 
to  provide  more  realistic  platform  movements  and 
tactically  appropriate  network  loads. 

•  Development  and  vahdation  of  a  test  bed  perfor¬ 
mance  model  is  needed  to  aid  in  the  generation  of 
the  performance  or  requirement  envelope. 

The  implementation  of  the  tactical  wireless  sensor 
network  test  bed  enhances  the  test  and  analysis  of 
system  performance  in  a  realistic  real-time,  high- 
fidelity  simulated  environment  not  achievable  through 
standard  test  processes.  The  test  bed  allows  the 
community  to  evaluate  large  tactical  NSUT  perfor¬ 
mance  parameters  such  as  throughput,  latency,  jitter, 
dropped  packets,  message  completion  rate,  channel 
interference,  jamming,  bottlenecks,  power  consump¬ 
tion,  and  reliability  just  to  name  a  few.  The  primary 
advantage  of  this  architecture  is  the  inclusion  of  live 
hardware  in  the  test,  which  wiU  be  immersed  in  an 
augmented  environment  that  allows  the  item  under 
test  to  perceive  and  respond  to  stimulus  just  as  it  would 
in  the  real  world.  Each  layer  of  the  network  can  be 
tested  because  of  the  high  fidelity  simulation  made 


possible  by  utilizing  parallel  processing  to  maintain 
real-time  performance.  □ 
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